test(data): source freshness on all bronze connectors (#1322) - #1346

No actionable comments were generated in the recent review. 🎉

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Coordinating — I have a ready "runtime/alerting" layer that builds directly on this PR.

This PR nails the declaration half of #1322 (per-source freshness blocks so dbt_coverage.py counts them toward 130/130). I've got the complementary runtime half already written against the old tree, and it sits cleanly on top of yours rather than competing with it:

• dbt-source-freshness-check CronWorkflow — runs dbt source freshness --select source:* daily after the sync window, parses target/sources.json, logs each breaching source with max_loaded_at + lag, and flips the workflow to Failed only on error/runtime error (warn stays green, still printed + emitted in payload).
• Driver-based notification (webhook / zulip / slack / teams) off a provider-agnostic JSON payload.
• Runtime trap-detector (freshness-trap-detect.py) for the "dbt couldn't even check" case (CH down, schema drift).
• PRD/DESIGN/FEATURE for a new ingestion-monitoring domain.

None of that exists in #1322's scope (which is declaration + coverage gate), so it's additive. Two things to align:

1. Thresholds — I had 30h/48h, you have 36h/72h. Happy to defer to yours; the cron reads whatever the sources declare, so no duplication.
2. Sequencing — my cron consumes the per-source declarations this PR adds, so it should land after this merges. I'd open it as a follow-up PR referencing EPIC: data-correctness contract coverage (bronze · silver · gold) → 100% #1321 as the runtime/alerting half of data(bronze): source freshness on all sources + contracts on *__bronze_promoted views #1322.

One design note for the record: my original took the inherited-default route (loaded_at_field + thresholds in dbt_project.yml, per-table opt-outs). That's functionally equivalent but wouldn't satisfy dbt_coverage.py, which counts per-source declarations — so your per-source approach is the right call and I'll build on it as-is.

Want me to open the follow-up PR once this merges, or fold the cron into this branch?

Thanks — agreed on all of it, and good that the runtime half is already written.

On the two alignment points:

1. Thresholds — keep 36h/72h here. The bronze syncs are daily, so 36h warn is roughly a 1.5-day grace (one missed sync isn't yet alarming) and 72h error covers a weekend gap before anyone gets paged. Since your cron reads whatever the sources declare, there's a single source of truth and no duplication — if we want to tighten later, it changes in one place.

2. Sequencing — open it as a separate follow-up after this merges, not folded in. This PR is deliberately declaration-only: static, no warehouse needed, mergeable as-is. Your runtime layer is a different concern with its own domain (ingestion-monitoring) and its own PRD/DESIGN, so keeping them apart keeps both reviewable and lets this gate land without waiting on the alerting design. The dependency direction is right too — your cron consumes these per-source blocks, so this lands first and yours sits on top.

For tracking: the runtime/alerting CronWorkflow is the "scheduled bronze freshness" item in the stabilization plan (the monitoring line, #949), so it slots in there as the operational counterpart to this declaration gate.

And confirmed your read on dbt_coverage.py — it counts per-source declarations, which is exactly why we went per-source instead of the dbt_project.yml inherited default. Please go ahead and open the follow-up once this merges.

Heads-up before this lands: flat _airbyte_extracted_at is false-green on the windowed/vendor-analytics connectors — verified on live data (Eternal installation).

The check correctly catches "sync fully stopped" for true incremental connectors. But for any connector whose API returns a fixed reporting window that the sync re-fetches every run (M365 Graph reports, ChatGPT Team, Claude Team), _airbyte_extracted_at is re-stamped on every successful sync regardless of whether the business data advanced — so it measures sync liveness, not data freshness.

Measured just now on the Eternal ClickHouse (now() - max(...)):

Right now M365 business data is 4.5 days stale on this install, but warn_after: 36h / error_after: 72h on _airbyte_extracted_at reports green. The incremental connectors (youtrack/jira) are fine — ext_age ≈ biz_age — which is exactly the discriminator.

This matters because the false-green sources are the AI-usage connectors that Insight's value story leans on. The connector config is misleading here too — M365 declares lookback_window: P0D, but the Graph getReport endpoint returns the whole reportPeriod window, so the data re-fetches regardless.

Proposal: adopt the per-source model from the earlier #256 work as the declaration here — it's demonstrably the correct instrument on exactly these connectors. Concretely, additive to this PR's per-source structure:

1. Anchor loaded_at_field on the business-date column for the windowed connectors instead of _airbyte_extracted_at — reportRefreshDate (M365), date (ChatGPT / Slack), metric_date (Claude). Incremental connectors (git, jira, youtrack) keep _airbyte_extracted_at as-is.
2. Tier the thresholds so the windowed connectors get a higher warn_after/error_after that absorbs the legitimate 2–3 day vendor publish lag — otherwise anchoring on the business date would false-alarm. (This is why bug: frontend calls /api/identity-resolution/* removed by PR #398 (regression) #256 carried tiers; the dogfood numbers above show the lag is real, not a bug.)
3. Per-table opt-outs (freshness: null + a written reason) for the slow-moving lookup/catalog streams.

I have the full per-connector loaded_at_field + tier mapping from #256 ready to go. Happy to push it straight onto this branch so the declaration half ships correct in one piece, or land it as the immediate follow-up the runtime PR sits on — whichever you prefer for review flow.

This is the right catch, and it changes my call from earlier in the thread — thank you for dogfooding it on live data.

You're right that _airbyte_extracted_at measures sync liveness, not data freshness, and that it goes false-green on exactly the windowed vendor-analytics connectors (M365 Graph, ChatGPT Team, Claude Team) where the API re-emits a fixed reporting window every run. I corroborated the anchors you propose are real columns on those connectors (reportRefreshDate on M365, metric_date on Claude, date on the ChatGPT/Slack streams), so the per-source anchoring is feasible as a drop-in.

So I'm reversing my "keep it flat, declaration-only is fine as-is" position: the per-source + tiered model belongs in this PR, not a follow-up. A freshness gate that reads green while M365 business data is 4.5 days stale is worse than no gate — and it's worst on the AI-usage connectors that matter most. My earlier 36h/72h reasoning only held for the true-incremental connectors (ext_age ≈ biz_age); it's wrong for the windowed ones.

Please push the #256 per-connector mapping straight onto this branch (claude/dbt-test-coverage) — you have the live-verified loaded_at_field + tier values and I'd only be guessing the per-stream business-date columns and the lag-absorbing thresholds, which is exactly where a wrong guess turns into false-alarm. Concretely, to land the declaration correct in one piece:

1. Anchor loaded_at_field on the business-date column for the windowed connectors; keep _airbyte_extracted_at for the incrementals (git/jira/youtrack) — ext_age ≈ biz_age is the clean discriminator.
2. Tiered warn_after/error_after so the windowed connectors absorb the real 2–3 day vendor publish lag (your numbers show it's genuine).
3. Per-table freshness: null + a one-line reason for the slow-moving lookup/catalog streams.

The runtime/alerting CronWorkflow stays the separate follow-up we discussed — this keeps that PR sitting on a declaration that's actually correct. If pushing to the branch is awkward from your side, open it as a PR into claude/dbt-test-coverage and I'll merge it in.

Opened it as a PR into this branch: SharedQA#1 — anchors the windowed connectors (m365 / chatgpt_team / claude_team verified, the rest column-verified or inferred-and-marked) on their business-date column with tiered thresholds; incrementals untouched. The PR body breaks down the three confidence tiers and the one caveat (no end-to-end dbt source freshness run on the full set yet). Ready for you to merge in whenever it looks right.

One more before this lands — reviewing the merged anchors I caught a latent bug and opened SharedQA#2 against this branch:

• openai: bucket_start_time is Unix seconds (integer), not a date string — the connector's own models use fromUnixTimestamp(CAST(... AS UInt32)). The anchor I added in [PRE-01] Define Priority-1 connectors for VZ #1 used parseDateTimeBestEffortOrNull, which would runtime error on every openai usage/cost table once it's ingested. [PRE-02] Define Priority-1 charts and dashboards #2 fixes it to the epoch form.
• confluence/outline: aligned wiki_pages to parseDateTime64BestEffortOrNull(updated_at, 3) to match each connector's own model convention (functionally equivalent, just consistent).

openai is empty in the dogfood install so it wouldn't fail today, but it's a latent false-positive. Worth folding #2 in before #1346 merges so the gate ships correct. All other inferred anchors were re-checked against the connectors' silver models and parse fine.